Predictive RF beamforming for head mounted display

ABSTRACT

A method is provided, including the following method operations: receiving captured images of an interactive environment in which a head-mounted display (HMD) is disposed; receiving inertial data processed from at least one inertial sensor of the HMD; analyzing the captured images and the inertial data to determine a predicted future location of the HMD; using the predicted future location of the HMD to adjust a beamforming direction of an RF transceiver towards the predicted future location of the HMD.

BACKGROUND 1. Field of the Disclosure

The present disclosure relates to predictive RF beamforming fortransmission of data to head mounted displays (HMDs), and relatedmethods, apparatus, and systems.

2. Description of the Related Art

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce very detailed and engaging gaming experiences.

Example gaming platforms include the Sony Playstation®, SonyPlaystation2® (PS2), Sony Playstation3® (PS3), and Sony Playstation4®(PS4), each of which is sold in the form of a game console. As is wellknown, the game console is designed to connect to a display (typically atelevision) and enable user interaction through handheld controllers.The game console is designed with specialized processing hardware,including a CPU, a graphics synthesizer for processing intensivegraphics operations, a vector unit for performing geometrytransformations, and other glue hardware, firmware, and software. Thegame console may be further designed with an optical disc reader forreceiving game discs for local play through the game console. Onlinegaming is also possible, where a user can interactively play against orwith other users over the Internet. As game complexity continues tointrigue players, game and hardware manufacturers have continued toinnovate to enable additional interactivity and computer programs.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between the user and the gaming system. One wayof accomplishing a richer interactive experience is to use wireless gamecontrollers whose movement is tracked by the gaming system in order totrack the player's movements and use these movements as inputs for thegame. Generally speaking, gesture input refers to having an electronicdevice such as a computing system, video game console, smart appliance,etc., react to some gesture made by the player and captured by theelectronic device.

Another way of accomplishing a more immersive interactive experience isto use a head-mounted display. A head-mounted display is worn by theuser and can be configured to present various graphics, such as a viewof a virtual space. The graphics presented on a head-mounted display cancover a large portion or even all of a user's field of view. Hence, ahead-mounted display can provide a visually immersive experience to theuser.

A head-mounted display (HMD) provides an immersive virtual realityexperience, as the HMD renders a real-time view of the virtualenvironment in a manner that is responsive to the user's movements. Theuser wearing an HMD is afforded freedom of movement in all directions,and accordingly can be provided a view of the virtual environment in alldirections via the HMD. However, the processing resources required togenerate the video for rendering on the HMD are considerable andtherefore handled by a separate computing device, such as a personalcomputer or a game console. The computing device generates the video forrendering to the HMD, and transmits the video to the HMD.

To provide a high fidelity experience, it is desirable to provide highquality video (e.g. at high resolution and frame rate). However, suchvideo entails transmission of large amounts of data, requiring highbandwidth and a stable connection. Thus, current systems for HMDrendering use a wired connection to transfer data from the computingdevice to the HMD, as this affords the requisite bandwidth andconnection stability. However, the presence of a wire that connects tothe HMD can be bothersome to the user, as it may contact the user anddetract from the immersive experience of using the HMD. Furthermore, thewired connection may inhibit the user's freedom of movement, as the usermust be mindful of not over-extending the wire, and must avoid anymovement which might cause disconnection or damage the wire.Furthermore, the presence of the wire presents a tripping hazard, whichis amplified by the fact that the user cannot see the real environmentwhile using the HMD.

It is in this context that implementations of the disclosure arise.

SUMMARY

Implementations of the present disclosure include devices, methods andsystems relating to RF beamforming for a head mounted display.

In some implementations, a method is provided, including the followingmethod operations: receiving captured images of an interactiveenvironment in which a head-mounted display (HMD) is disposed; receivinginertial data processed from at least one inertial sensor of the HMD;analyzing the captured images and the inertial data to determine apredicted future location of the HMD; using the predicted futurelocation of the HMD to adjust a beamforming direction of an RFtransceiver towards the predicted future location of the HMD.

In some implementations, analyzing the captured images and the inertialdata includes identifying movement of the HMD, the predicted futurelocation of the HMD being determined using the identified movement ofthe HMD.

In some implementations, identifying movement of the HMD includesdetermining a motion vector of the HMD, the predicted future location ofthe HMD being determined by applying the motion vector of the HMD to acurrent location of the HMD.

In some implementations, a magnitude of the motion vector identifies aspeed of the movement of the HMD, and wherein a direction of the motionvector identifies a direction of the movement of the HMD.

In some implementations, the method further includes: adjusting anangular spread of the RF transceiver based on the speed of the movementof the HMD.

In some implementations, the angular spread increases with increasingspeed of the movement of the HMD.

In some implementations, identifying movement of the HMD includesidentifying translational movement and/or rotational movement of theHMD; wherein determining the motion vector includes determining anacceleration of the translational movement and/or the rotationalmovement.

In some implementations, the RF transceiver includes a phased array ofRF emitters; wherein adjusting the beamforming direction of the RFtransceiver includes generating transceiver control data that isconfigured to cause adjustment of a phase or amplitude of at least oneof the RF emitters of the phased array.

In some implementations, the at least one inertial sensor of the HMDincludes one or more of an accelerometer, a gyroscope, or amagnetometer.

In some implementations, the HMD includes a plurality of lights; whereinanalyzing the captured images includes identifying one or more of theplurality of lights in the captured images.

In some implementations, a system is provided, including: a head-mounteddisplay (HMD), the HMD having at least one inertial sensor configured togenerate inertial data; a camera configured to capture images of aninteractive environment in which the HMD is disposed; an RF transceiver;a computer configured to analyze the captured images and the inertialdata to determine a predicted future location of the HMD, and use thepredicted future location of the HMD to adjust a beamforming directionof the RF transceiver towards the predicted future location of the HMD.

Other aspects and advantages of the disclosure will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principles ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may be better understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 illustrates a system for interaction with a virtual environmentvia a head-mounted display (HMD), in accordance with an embodiment ofthe disclosure.

FIGS. 2A-1 and 2A-2 illustrate a head-mounted display (HMD), inaccordance with an embodiment of the disclosure.

FIG. 2B illustrates one example of an HMD user interfacing with a clientsystem, and the client system providing content to a second screendisplay, which is referred to as a second screen, in accordance with oneembodiment.

FIG. 3 conceptually illustrates the function of an HMD in conjunctionwith an executing video game, in accordance with an embodiment of thedisclosure.

FIG. 4 illustrates adjustment of a beamforming direction of atransceiver based on prediction of a future location of an HMD, inaccordance with implementations of the disclosure.

FIGS. 5A and 5B illustrate adjustment of the beamforming angular spreadbased on HMD movement, in accordance with implementations of thedisclosure.

FIG. 5C is a graph illustrating beamforming angular spread of atransceiver versus speed of an HMD, in accordance with implementationsof the disclosure.

FIG. 5D is a graph illustrating beamforming angular spread of atransceiver versus radial distance of the HMD from the transceiver, inaccordance with implementations of the disclosure.

FIG. 5E is graph illustrating beamforming angular spread of atransceiver versus transmission data rate, in accordance withimplementations of the disclosure.

FIGS. 6A, 6B, and 6C illustrate a scenario wherein the beamformingdirection is adjusted based on the gaze direction of the user 100, inaccordance with implementations of the disclosure.

FIG. 7 illustrates an overhead view of a room 700 showing locationdistribution of an HMD, in accordance with implementations of thedisclosure.

FIG. 8 conceptually illustrates the use of a prediction model todetermine beamforming parameters, in accordance with implementations ofthe disclosure.

FIG. 9 illustrates a method for adjusting beamforming parameters using apredicted future location, in accordance with implementations of thedisclosure.

FIG. 10 conceptually illustrates a system for providing wirelesscommunication between a computer and a HMD, in accordance withimplementations of the disclosure.

FIG. 11A is a schematic diagram showing components of a beamformingtransmitter, in accordance with implementations of the disclosure.

FIG. 11B is a schematic diagram showing components of a beamformingreceiver, in accordance with implementations of the disclosure.

FIG. 12A conceptually illustrates a HMD having a plurality of antennaarrays, in accordance with implementations of the disclosure.

FIGS. 12B, 12C, and 12D illustrate overhead views of an HMD in aninteractive real environment, illustrating switching of active antennaarrays on an HMD, in accordance with implementations of the disclosure.

FIG. 13 illustrates components of a head-mounted display, in accordancewith an embodiment of the disclosure.

FIG. 14 is a block diagram of a Game System 1400, according to variousembodiments of the disclosure.

DETAILED DESCRIPTION

The following implementations of the present disclosure provide devices,methods, and systems relating to predictive RF beamforming for a headmounted display (HMD).

In various implementations, the methods, systems, image capture objects,sensors and associated interface objects (e.g., gloves, controllers,peripheral devices, etc.) are configured to process data that isconfigured to be rendered in substantial real time on a display screen.The display may be the display of a head mounted display (HMD), adisplay of a second screen, a display of a portable device, a computerdisplay, a display panel, a display of one or more remotely connectedusers (e.g., whom may be viewing content or sharing in an interactiveexperience), or the like.

It will be obvious, however, to one skilled in the art, that the presentdisclosure may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentdisclosure.

FIG. 1 illustrates a system for interaction with a virtual environmentvia a head-mounted display (HMD), in accordance with an embodiment ofthe disclosure. A user 100 is shown wearing a head-mounted display (HMD)102. The HMD 102 is worn in a manner similar to glasses, goggles, or ahelmet, and is configured to display a video game or other content tothe user 100. The HMD 102 provides a very immersive experience to theuser by virtue of its provision of display mechanisms in close proximityto the user's eyes. Thus, the HMD 102 can provide display regions toeach of the user's eyes which occupy large portions or even the entiretyof the field of view of the user.

In the illustrated embodiment, the HMD 102 is wirelessly connected to acomputer 106. The computer 106 can be any general or special purposecomputer known in the art, including but not limited to, a gamingconsole, personal computer, laptop, tablet computer, mobile device,cellular phone, tablet, thin client, set-top box, media streamingdevice, etc. In one embodiment, the computer 106 can be configured toexecute a video game, and output the video and audio from the video gamefor rendering by the HMD 102. A transceiver 110 is configured towirelessly transmit the video and audio from the video game to the HMD102 for rendering thereon. The transceiver 110 includes a transmitterfor wireless transmission of data to the HMD 102, as well as a receiverfor receiving data that is wirelessly transmitted by the HMD 102.

In some implementations, the HMD 102 may also communicate with thecomputer through alternative mechanisms or channels, such as via anetwork 112 to which both the HMD 102 and the computer 106 areconnected.

The user 100 may operate an interface object 104 to provide input forthe video game. Additionally, a camera 108 can be configured to captureimages of the interactive environment in which the user 100 is located.These captured images can be analyzed to determine the location andmovements of the user 100, the HMD 102, and the interface object 104. Invarious implementations, the interface object 104 includes a light whichcan be tracked, and/or inertial sensor(s), to enable determination ofthe interface object's location and orientation.

The way the user interfaces with the virtual reality scene displayed inthe HMD 102 can vary, and other interface devices in addition tointerface object 104, can be used. For instance, various kinds ofsingle-handed, as well as two-handed controllers can be used. In someembodiments, the controllers can be tracked themselves by trackinglights associated with the controllers, or tracking of shapes, sensors,and inertial data associated with the controllers. Using these varioustypes of controllers, or even simply hand gestures that are made andcaptured by one or more cameras, it is possible to interface, control,maneuver, interact with, and participate in the virtual realityenvironment presented on the HMD 102.

Additionally, the HMD 102 may include one or more lights which can betracked to determine the location and orientation of the HMD 102. Thecamera 108 can include one or more microphones to capture sound from theinteractive environment. Sound captured by a microphone array may beprocessed to identify the location of a sound source. Sound from anidentified location can be selectively utilized or processed to theexclusion of other sounds not from the identified location. Furthermore,the camera 108 can be defined to include multiple image capture devices(e.g. stereoscopic pair of cameras), an IR camera, a depth camera, andcombinations thereof.

In another embodiment, the computer 106 functions as a thin client incommunication over a network 112 with a cloud gaming provider 114. Insuch an implementation, generally speaking, the cloud gaming provider114 maintains and executes the video game being played by the user 102.The computer 106 transmits inputs from the HMD 102, the directionalinterface object 104 and the camera 108, to the cloud gaming provider,which processes the inputs to affect the game state of the executingvideo game. The output from the executing video game, such as videodata, audio data, and haptic feedback data, is transmitted to thecomputer 106. The computer 106 may further process the data beforetransmission or may directly transmit the data to the relevant devices.For example, video and audio streams are provided to the HMD 102,whereas a vibration feedback command is provided to the interface object104.

In some embodiments, the HMD 102, interface object 104, and camera 108,may themselves be networked devices that connect to the network 112, forexample to communicate with the cloud gaming provider 114. In someimplementations, the computer 106 may be a local network device, such asa router, that does not otherwise perform video game processing, butwhich facilitates passage of network traffic. The connections to thenetwork by the HMD 102, interface object 104, and camera 108 may bewired or wireless.

Additionally, though embodiments in the present disclosure may bedescribed with reference to a head-mounted display, it will beappreciated that in other embodiments, non-head mounted displays may besubstituted, including without limitation, portable device screens (e.g.tablet, smartphone, laptop, etc.) or any other type of display that canbe configured to render video and/or provide for display of aninteractive scene or virtual environment in accordance with the presentembodiments.

The amount of data, especially in the form of video data (e.g. includingimage data and audio data), that must be transmitted to the HMD toprovide a high quality user experience when viewing a virtualenvironment is quite large. For this reason, current HMD technologyrequires a wired connection between the computer which generates thevideo data, and the HMD. However, as noted above, a wired connection tothe HMD detracts from the user's freedom of movement, degrading theotherwise immersive experience that can be so effectively renderedthrough an HMD.

Providing a wireless connection that is capable of reliably transmittingthe amount of data required for a high quality experience requiresovercoming problems in terms of providing a high signal-to-noise ratiofor high data bandwidth while also maintaining high connection stabilityto the HMD as it moves in accordance with movements of the user. Toaccomplish this, implementations of the present disclosure provide forwireless data transmission to the HMD using predictive beamforming. Thatis, in some implementations, tracked movement of the HMD is analyzed topredict future locations of the HMD, and beamforming is used topredictively steer an RF signal towards the predicted future locationsof the HMD. RF signal strength is thereby maintained by steering the RFsignal in an anticipatory manner so as to better track the HMD'slocation.

For purposes of ease of description in the present disclosure, referenceis made to the actual or predicted location of the HMD as a location inthe real-world space towards which an RF signal should be directed.However, it should be appreciated that the location of the HMD may morespecifically refer to a particular location on, within, or relative to,the HMD, such as the location of a receiver antenna that is part of theHMD, a location of the display portion of the HMD, a center of the HMD,etc.

With continued reference to FIG. 1, an overview of a procedure forpredictive beamforming for data transmission to an HMD is shown, inaccordance with implementations of the disclosure. It should beappreciated that the location of the HMD 102 can be tracked using anyvariety of technologies. In the illustrated implementation, the HMD 102transmits inertial data 116 generated from one or more inertial sensorsof the HMD to the computer 106. Further, the computer 106 receivescaptured image data 118 from the camera 110, which is configured tocapture images of the interactive environment in which the HMD 102 andthe user 100 are disposed. The inertial data 116 and/or the image data118 are analyzed by the computer 106 to identify and track the HMD 102and its location, orientation, and movements.

A predicted future location of the HMD is determined using the trackedmovements of the HMD 102. By way of example, a motion vector can begenerated by the computer 106 based on the tracked movements of the HMD102. This motion vector can be applied to the current location of theHMD 102 to predict the future location of the HMD. Using the predictedfuture location of the HMD, the computer 106 generates beamforming data120 that is configured to direct the beamforming direction of thetransceiver 110 towards the predicted future location of the HMD. Bydirecting the beamforming direction of the transceiver in a predictivemanner, a strong wireless signal can be maintained, as the movements ofthe HMD 102 will be anticipated and the beamforming direction of thesignal will not lag such movements, but can move in a simultaneousand/or anticipatory manner with such movements of the HMD. In thepresent disclosure, reference is made to the beamforming parameters(e.g. direction and angular spread) of the transceiver 110. It will beappreciated that such beamforming parameters can be applied to either orboth of the transmitter and the receiver which are parts of thetransceiver. Broadly speaking, implementations focused on transmissionof video data from the computer to the HMD may discuss beamforming interms of transmission by the transceiver's transmitter. However, itshould be appreciated that any such discussion of beamforming can alsobe applied to signal reception by the transceiver's receiver.

In some implementations, the camera 108 and the transceiver 110 areintegrated in the same device, so that the camera and transceiver have afixed spatial relationship to each other, and more specifically, theimage capture by the camera and the RF beamforming by the transceiverare spatially known in relation to each other. In such implementations,the position of the HMD can be determined from captured images by thecamera, and the beamforming by the transceiver can be appropriatelydirected towards the HMD without additional calibration being required.

In other implementations, the transceiver 110 and the camera 108 areseparate devices which can be positioned in the local environment atdifferent locations. In such implementations, a calibration may beperformed to determine the spatial relationship of the image capture bythe camera and the RF beamforming by the transceiver. In oneimplementation, this can be performed by analyzing captured images fromthe camera to determine the location of the HMD relative to the camera,and performing a test to determine the optimal beamforming direction forthe determined location of the HMD, and correlating these pieces ofinformation. Such a procedure may be performed for multiple locations ofthe HMD to achieve more accurate calibration results.

In some implementations, signal quality feedback 122 is provided fromthe HMD 102 to the computer 106, e.g. via the transceiver 110 or thenetwork 112. The signal quality feedback 122 is indicative of thequality of the wireless transmission (e.g. signal strength, error rate,etc.), and provides information which can be used to evaluate whetherthe beamforming direction is being effectively steered towards the HMD102 so as to provide sufficient data transmission rates.

FIGS. 2A-1 and 2A-2 illustrate a head-mounted display (HMD), inaccordance with an embodiment of the disclosure. FIG. 2A-1 in particularillustrates the Playstation® VR headset, which is one example of a HMDin accordance with implementations of the disclosure. As shown, the HMD102 includes a plurality of lights 200A-H. Each of these lights may beconfigured to have specific shapes, and can be configured to have thesame or different colors. The lights 200A, 200B, 200C, and 200D arearranged on the front surface of the HMD 102. The lights 200E and 200Fare arranged on a side surface of the HMD 102. And the lights 200G and200H are arranged at corners of the HMD 102, so as to span the frontsurface and a side surface of the HMD 102. It will be appreciated thatthe lights can be identified in captured images of an interactiveenvironment in which a user uses the HMD 102. Based on identificationand tracking of the lights, the location and orientation of the HMD 102in the interactive environment can be determined. It will further beappreciated that some of the lights may or may not be visible dependingupon the particular orientation of the HMD 102 relative to an imagecapture device. Also, different portions of lights (e.g. lights 200G and200H) may be exposed for image capture depending upon the orientation ofthe HMD 102 relative to the image capture device.

In one embodiment, the lights can be configured to indicate a currentstatus of the HMD to others in the vicinity. For example, some or all ofthe lights may be configured to have a certain color arrangement,intensity arrangement, be configured to blink, have a certain on/offconfiguration, or other arrangement indicating a current status of theHMD 102. By way of example, the lights can be configured to displaydifferent configurations during active gameplay of a video game(generally gameplay occurring during an active timeline or within ascene of the game) versus other non-active gameplay aspects of a videogame, such as navigating menu interfaces or configuring game settings(during which the game timeline or scene may be inactive or paused). Thelights might also be configured to indicate relative intensity levels ofgameplay. For example, the intensity of lights, or a rate of blinking,may increase when the intensity of gameplay increases. In this manner, aperson external to the user may view the lights on the HMD 102 andunderstand that the user is actively engaged in intense gameplay, andmay not wish to be disturbed at that moment.

The HMD 102 may additionally include one or more microphones. In theillustrated embodiment, the HMD 102 includes microphones 204A and 204Bdefined on the front surface of the HMD 102, and microphone 204C definedon a side surface of the HMD 102. By utilizing an array of microphones,sound from each of the microphones can be processed to determine thelocation of the sound's source. This information can be utilized invarious ways, including exclusion of unwanted sound sources, associationof a sound source with a visual identification, etc.

The HMD 102 may also include one or more image capture devices. In theillustrated embodiment, the HMD 102 is shown to include image capturedevices 202A and 202B. By utilizing a stereoscopic pair of image capturedevices, three-dimensional (3D) images and video of the environment canbe captured from the perspective of the HMD 102. Such video can bepresented to the user to provide the user with a “video see-through”ability while wearing the HMD 102. That is, though the user cannot seethrough the HMD 102 in a strict sense, the video captured by the imagecapture devices 202A and 202B (e.g., or one or more front facing cameras108′ disposed on the outside body of the HMD 102, as shown in FIG. 3below) can nonetheless provide a functional equivalent of being able tosee the environment external to the HMD 102 as if looking through theHMD 102. Such video can be augmented with virtual elements to provide anaugmented reality experience, or may be combined or blended with virtualelements in other ways. Though in the illustrated embodiment, twocameras are shown on the front surface of the HMD 102, it will beappreciated that there may be any number of externally facing camerasinstalled on the HMD 102, oriented in any direction. For example, inanother embodiment, there may be cameras mounted on the sides of the HMD102 to provide additional panoramic image capture of the environment.

FIG. 2B illustrates one example of an HMD 102 user 100 interfacing witha client system 106, and the client system 106 providing content to asecond screen display, which is referred to as a second screen 207. Theclient system 106 may include integrated electronics for processing thesharing of content from the HMD 102 to the second screen 207. Otherembodiments may include a separate device, module, connector, that willinterface between the client system and each of the HMD 102 and thesecond screen 207. In this general example, user 100 is wearing HMD 102and is playing a video game using a controller, which may also bedirectional interface object 104. The interactive play by user 100 willproduce video game content (VGC), which is displayed interactively tothe HMD 102.

In one embodiment, the content being displayed in the HMD 102 is sharedto the second screen 207. In one example, a person viewing the secondscreen 207 can view the content being played interactively in the HMD102 by user 100. In another embodiment, another user (e.g. player 2) caninteract with the client system 106 to produce second screen content(SSC). The second screen content produced by a player also interactingwith the controller 104 (or any type of user interface, gesture, voice,or input), may be produced as SSC to the client system 106, which can bedisplayed on second screen 207 along with the VGC received from the HMD102.

Accordingly, the interactivity by other users who may be co-located orremote from an HMD user can be social, interactive, and more immersiveto both the HMD user and users that may be viewing the content played bythe HMD user on a second screen 207. As illustrated, the client system106 can be connected to the Internet 210. The Internet can also provideaccess to the client system 106 to content from various content sources220. The content sources 220 can include any type of content that isaccessible over the Internet.

Such content, without limitation, can include video content, moviecontent, streaming content, social media content, news content, friendcontent, advertisement content, etc. In one embodiment, the clientsystem 106 can be used to simultaneously process content for an HMDuser, such that the HMD is provided with multimedia content associatedwith the interactivity during gameplay. The client system 106 can thenalso provide other content, which may be unrelated to the video gamecontent to the second screen. The client system 106 can, in oneembodiment receive the second screen content from one of the contentsources 220, or from a local user, or a remote user.

FIG. 3 conceptually illustrates the function of the HMD 102 inconjunction with an executing video game, in accordance with anembodiment of the disclosure. The executing video game is defined by agame engine 320 which receives inputs to update a game state of thevideo game. The game state of the video game can be defined, at least inpart, by values of various parameters of the video game which definevarious aspects of the current gameplay, such as the presence andlocation of objects, the conditions of a virtual environment, thetriggering of events, user profiles, view perspectives, etc.

In the illustrated embodiment, the game engine receives, by way ofexample, controller input 314, audio input 316 and motion input 318. Thecontroller input 314 may be defined from the operation of a gamingcontroller separate from the HMD 102, such as a handheld gamingcontroller (e.g. Sony DUALSHOCK®4 wireless controller, Sony PlayStation®Move motion controller) or directional interface object 104. By way ofexample, controller input 314 may include directional inputs, buttonpresses, trigger activation, movements, gestures, or other kinds ofinputs processed from the operation of a gaming controller. The audioinput 316 can be processed from a microphone 302 of the HMD 102, or froma microphone included in the image capture device 108 or elsewhere inthe local environment. The motion input 318 can be processed from amotion sensor 300 included in the HMD 102, or from image capture device108 as it captures images of the HMD 102. The game engine 320 receivesinputs which are processed according to the configuration of the gameengine to update the game state of the video game. The game engine 320outputs game state data to various rendering modules which process thegame state data to define content which will be presented to the user.

In the illustrated embodiment, a video rendering module 322 is definedto render a video stream for presentation on the HMD 102. The videostream may be presented by a display/projector mechanism 310, and viewedthrough optics 308 by the eye 306 of the user. An audio rendering module304 is configured to render an audio stream for listening by the user.In one embodiment, the audio stream is output through a speaker 304associated with the HMD 102. It should be appreciated that speaker 304may take the form of an open air speaker, headphones, or any other kindof speaker capable of presenting audio.

In one embodiment, a gaze tracking camera 312 is included in the HMD 102to enable tracking of the gaze of the user. The gaze tracking cameracaptures images of the user's eyes, which are analyzed to determine thegaze direction of the user. In one embodiment, information about thegaze direction of the user can be utilized to affect the videorendering. For example, if a user's eyes are determined to be looking ina specific direction, then the video rendering for that direction can beprioritized or emphasized, such as by providing greater detail or fasterupdates in the region where the user is looking. It should beappreciated that the gaze direction of the user can be defined relativeto the head mounted display, relative to a real environment in which theuser is situated, and/or relative to a virtual environment that is beingrendered on the head mounted display.

Broadly speaking, analysis of images captured by the gaze trackingcamera 312, when considered alone, provides for a gaze direction of theuser relative to the HMD 102. However, when considered in combinationwith the tracked location and orientation of the HMD 102, a real-worldgaze direction of the user can be determined, as the location andorientation of the HMD 102 is synonymous with the location andorientation of the user's head. That is, the real-world gaze directionof the user can be determined from tracking the positional movements ofthe user's eyes and tracking the location and orientation of the HMD102. When a view of a virtual environment is rendered on the HMD 102,the real-world gaze direction of the user can be applied to determine avirtual world gaze direction of the user in the virtual environment.

Additionally, a tactile feedback module 326 is configured to providesignals to tactile feedback hardware included in either the HMD 102 oranother device operated by the user, such as directional interfaceobject 104. The tactile feedback may take the form of various kinds oftactile sensations, such as vibration feedback, temperature feedback,pressure feedback, etc. The directional interface object 104 can includecorresponding hardware for rendering such forms of tactile feedback.

FIG. 4 illustrates adjustment of a beamforming direction of atransmitter based on prediction of a future location of an HMD, inaccordance with implementations of the disclosure. In the illustratedimplementation, the HMD 102 is shown in a three-dimensional space at aninitial location A. The HMD 102 is capable of being moved in anydirection under the control of a user, and as such it is desirable tosteer the transmission beam towards the HMD 102.

In some implementations, a motion vector 400 is determined that isindicative of the current movement of the HMD 102. The current movementof the HMD can be determined from data generated by one or more inertialsensors of the HMD 102, as well as from analyzing captured images of theHMD (e.g. to track movement of lights or other recognizable portions ofthe HMD). In some implementations, the motion vector 400 is a velocityvector indicating both a spatial (three-dimensional (3D)) direction ofthe HMD's movement and a speed of the movement. The motion vector 400can be applied to the current location A of the HMD to determine apredicted future location B of the HMD. That is, the future location Bis predicted by extrapolating from the current location A using thedirection and speed of movement of the HMD.

In some implementations, the motion vector 400 is itself predicted basedon a determined acceleration of the HMD 102. That is the change in thevelocity (including changes in the direction and speed) of the HMD canbe determined from previously determined velocities of the HMD atearlier time points, and/or acceleration-sensing hardware (e.g. one ormore accelerometers) defining the current acceleration of the HMD. Thisacceleration can be applied to the immediately preceding motion vectorto determine the motion vector 400, which is applied to the currentlocation to predict the future location as described above.

In the illustrated implementation, the initial beamforming direction 402of the transceiver 110 is directed towards the initial location A of theHMD as shown. Based on the predicted future location B of the HMD, thebeamforming direction is adjusted so as to be directed towards thefuture location B, as indicated by the updated beamforming direction404. It will be appreciated that the adjustment of the beamformingdirection is performed in a predictive manner that occurs before theactual future location of the HMD 102 is known. By anticipating thefuture location of the HMD, and predictively steering the beamformingdirection accordingly, the wireless communication between thetransceiver 110 and the HMD 102 can be improved, as the improvedbandwidth that is provided via RF beamforming is maintained bycontinually steering its direction towards the HMD 102.

It will be appreciated that the beamforming direction is predictivelyadjusted, and therefore may or may not match the actual movement of theHMD to various extents. However, in accordance with implementations ofthe disclosure, a subsequent predicted location can be determined from aknown current location that is determined based on the latest availableinformation (e.g. via analysis of captured images from the camera).Thus, although a given adjusted beamforming direction may notspecifically match the actual movement of the HMD, a subsequentadjustment of the beamforming direction will be based, at least in part,on the actual known location of the HMD, and therefore, the continualadjustment of the beamforming direction will not be susceptible toexcessive deviation from the actual location of the HMD 102.

In some implementations, the beamforming update rate is on the order ofabout 10 to 100 milliseconds, and therefore the rate at which the futurelocation of the HMD is predicted matches that of the beamforming updaterate. In some implementations, the prediction rate is configured tomatch the frame rate of the camera, e.g. 60, 120, or 240 Hz in someimplementations. Thus, the prediction will be to predict the location ofthe HMD at the next frame.

In some implementations, the inertial sensors of the HMD 102 may havebetter capabilities for detecting movement than the camera 108. Forexample, the inertial sensors may be sensitive to smaller movements thanthe camera 108, as the camera may be limited by its resolution (e.g.720p or 1080p resolutions in some implementations). Furthermore, thesample rates of the inertial sensors may be significantly higher thanthe frame rate of the camera. For example, the camera may have a framerate of about 60, 120 or 240 Hz, while the inertial sensors may havesample rates of over 1000 Hz. Further, the camera may require greaterprocessing time (e.g. to analyze captured images) to determine locationand/or movement. Thus, the inertial sensors can be more sensitive tomovement with faster transient response that the camera 108.

However, the inertial sensors that detect relative movement can be proneto drift effects over time, and therefore are not exclusively reliedupon to provide determinations of HMD location. Whereas, the camera 108is better suited to provide accurate determinations of the location ofthe HMD, as fixed objects in the local environment can serve as anchorsfor purposes of determining the location of the HMD within the localenvironment.

Therefore, in various implementations, the use of inertial sensor dataversus image capture data, either separately or in combination, can varyover time. For example, in some implementations, the sample rate of theinertial sensors may be N times faster than the frame rate of thecamera. Thus, the predicted location of the HMD can be determined at arate matching the sample rate of the inertial sensors, but with everyNth predicted location taking into account the image capture data fromthe camera (e.g. to verify the actual location of the HMD, on the basisof which the predicted location is determined). It will be appreciatedthat with each predicted location of the HMD, the beamforming directionof the transceiver 110 can be adjusted accordingly so as to be directedtowards the predicted location of the HMD. Thus, the adjustments inbeamforming direction may occur at a faster rate than the frame rate ofthe camera.

In related implementations, the rate at which the predicted locations ofthe HMD are determined does not necessarily match the sample rate of theinertial sensors, but is nonetheless faster than the frame rate of thecamera, and/or faster than the rate at which predicted locationdeterminations take into account captured image data. It will beappreciated that the sample rates of the inertial sensors and framerates of the camera can be configurable within the operating ranges ofthese devices, and that such can be controlled as necessary to enablelocation prediction as discussed.

In some implementations, the faster sample rate of the inertial sensorsis leveraged to improve determinations of the motion vector, for exampleby taking into account the acceleration of the HMD in real space basedon the (additionally sampled, versus the captured images) inertialsensor data. The motion vector 400 may thus be better tailored to matchthe actual motion of the HMD, and thereby enable more accurate predictedlocations of the HMD.

In some implementations, the time required to process and analyzecaptured image data from the camera is such that determinations of HMDlocation using the captured image data may lag the actual movements ofthe HMD to a noticeable extent. Thus, in some implementations, thecaptured image data is analyzed to determine the HMD's historicallocation, but not utilized as the current location for purposes ofdetermining the predicted future location (based on inertial sensordata). Rather, the analysis of the captured image data is carried outand utilized to verify the historical location of the HMD, for example,against a previously predicted location of the HMD. The currentprediction of HMD location may be adjusted based on such information if,for example, the previously predicted location of the HMD differs fromthe historical location by greater than a predefined amount.

Additionally, as discussed in further detail below, the prediction ofHMD location may employ a prediction model. The accuracy of theprediction model may be evaluated based on comparing the historicallocation of the HMD, determined using the captured image data from thecamera, against a previously predicted location for the same time. Theprediction model may be adjusted based on such a comparison to provideimproved results.

FIGS. 5A and 5B illustrate adjustment of the beamforming angular spreadbased on HMD movement, in accordance with implementations of thedisclosure. It will be appreciated that in the present disclosure, thebeamforming direction refers to the peak intensity direction of the mainlobe of a beamforming transceiver 110. However, in addition to adjustingthe beamforming direction, the beamforming angular spread, which is theangular width/spread of the main lobe, can also be adjusted. The angularspread of an electromagnetic beam can be defined using variousdefinitions, such as the “full width at half maximum” (FWHM) (or “halfpower beam width” (HPBW) definition, which defines angular spread as thefull width of the beam at half its maximum intensity.

In some implementations, the angular spread is adjusted based on thespeed of the HMD 102. For example, at FIG. 5A, the HMD 102 operated byuser 100 has a first speed indicated by the motion vector 500.Accordingly, the beamforming angular spread of the transceiver 110 iscontrolled to have an angular spread 502. At FIG. 5B, the HMD 102operated by user 100 has a second speed indicated by the motion vector504, which is faster than the first speed. Accordingly, the beamformingangular spread of the transceiver 110 is controlled to have an angularspread 506, which is wider/greater than the angular spread 502. Thepresently described implementation contemplates adjustment of thebeamforming spread in manner that is positively correlated to the speedof the HMD, such that angular spread increases as HMD speed increases.This is useful for maintaining wireless connection stability, as therange of possible future locations of the HMD may tend to be greaterwhen the HMD's speed is higher, and therefore a beamforming angularspread having greater angular width under such circumstances is morelikely to maintain the HMD within the spread of the main lobe.

In a related implementation, the lateral speed of the HMD relative tothe transceiver is prioritized versus the speed of the HMD in otherdirections, for purposes of determining the beamforming angular spread.It will be appreciated that when the HMD 102 is moving towards or awayfrom the transceiver 110, the HMD may be less likely to move out of themain lobe of the transceiver, as opposed to when the HMD is moving in alateral direction relative to the transceiver. Therefore, in someimplementations, lateral movement of the HMD 102 relative to thetransceiver 110 is considered, and the beamforming angular spread isadjusted in a positive correlation to the lateral speed.

In some implementations, the beamforming angular spread of thetransceiver 110 is adjusted as a function of lateral speed of the HMDrelative to the transceiver, to the exclusion of HMD speed in othernon-lateral directions, such that the angular spread increases aslateral speed increases. In other implementations, the beamformingangular spread of the transceiver 110 is adjusted as a function of speedof the HMD, in a positive correlation such that angular spread increasesas HMD speed increases, but with the lateral speed of the HMD beingweighted more than HMD speed in other directions for purposes ofdetermining the angular spread.

In some implementations, the distance of the HMD from the transceiveraffects the beamforming angular spread. For example, when the HMD iscloser to the transceiver, then movements of the HMD may be more likelyto move the HMD out of the main lobe of the transceiver, versus when theHMD is further from the transceiver. Therefore, in some implementations,the beamforming angular spread is adjusted in inverse correlation todistance of the HMD from the transceiver, such that the angular spreadincreases as distance of the HMD from the transceiver decreases.

In related implementations, the concept can be applied based on detectedmovements of the HMD. For example, in some implementations, thebeamforming angular spread is adjusted based on radial movement of theHMD towards/away from the transceiver, such that the angular spread isincreased when radial movement of the HMD towards the transceiver isdetected, and the angular spread is decreased when radial movement ofthe HMD away from the transceiver is detected. Furthermore, the amountof the increase or decrease in angular spread can be positivelycorrelated to the speed of the HMD's radial movement towards or awayfrom the transceiver, respectively.

FIG. 5C is a graph illustrating beamforming angular spread of atransceiver versus speed of an HMD, in accordance with implementationsof the disclosure. Broadly speaking, the angular spread is positivelycorrelated to the speed of the HMD, such that as HMD speed increases, sodoes the angular spread of the transceiver. However, below a certainminimum speed, the angular spread is maintained at a minimum value. Andabove a certain maximum speed, the angular spread is maintained at amaximum value. In some implementations, the speed of the HMD isspecifically the lateral speed of the HMD relative to the transceiver.It will be appreciated that in accordance with the principles of thepresent disclosure, the speed of the HMD may be a predicted speed, e.g.based on factors such as a current speed and/or acceleration, and thatthe adjustment of the angular spread based on speed can thus beperformed in a predictive manner.

FIG. 5D is a graph illustrating beamforming angular spread of atransceiver versus radial distance of the HMD from the transceiver. Asshown, the angular spread generally inversely correlated to the radialdistance of the HMD from the transceiver, with angular spread generallydecreasing as the radial distance increases. However, below a certainminimum radial distance, the angular spread is maintained at a maximumvalue. And above a certain maximum radial distance the angular spread ismaintained at a minimum value. It will be appreciated that in accordancewith the principles of the present disclosure, the radial distance ofthe HMD from the transceiver may be a predicted radial distance, e.g.based on various factors such as current movement and acceleration, andthat the adjustment of the angular spread based on radial distance canthus be performed in a predictive manner.

In some implementations, the angular spread can be determined based onother factors, such as data rate. FIG. 5E is graph illustratingbeamforming angular spread of a transceiver versus transmission datarate, in accordance with implementations of the disclosure. Broadlyspeaking, the angular spread is inversely correlated to the transmissiondata rate, so that angular spread decreases as the data rate increases.A narrower angular spread can provide higher bandwidth, albeit over anarrower width. Thus, by changing the angular spread as a function ofdata rate in this manner, there is a tradeoff between the availablebandwidth when the signal is properly directed towards the HMD, and thewireless connection's tolerance to movement of the HMD. In someimplementations, below a certain minimum data rate, the angular spreadis maintained at a maximum value. And above a certain maximum data rate,the angular spread is maintained at a minimum value.

The above-described implementations which relate to adjustment of thebeamforming angular spread are provided by way of example, withoutlimitation. Further implementations falling within the scope of thepresent disclosure are encompassed by the combination of any of theforegoing implementations which are not exclusive of each other.

In some implementations, the beamforming direction and/or angular spreadcan be adjusted based on the gaze direction of the user. FIGS. 6A, 6B,and 6C illustrate a scenario wherein the beamforming direction isadjusted based on the gaze direction of the user 100, in accordance withimplementations of the disclosure. FIG. 6A shows an overhead view of theuser 100 wearing the HMD 102. The user 100 is shown having a gazedirection 600. The transceiver 110 is configured to have a beamformingdirection 602 that is directed towards the HMD 102. It will beappreciated that the angular spread of the transceiver is approximatelycentered about the HMD 102.

At FIG. 6B, the user 100 has moved his gaze direction to the right to agaze direction 606. A change in the gaze direction of the user 100 maybe indicative that the user is about to move, for example, approximatelyin the direction of the new gaze direction. Therefore, in accordancewith some implementations, the beamforming direction of the transceiver110 is adjusted in response to changes in the user's gaze direction.With continued reference to FIG. 6B, as the gaze direction 606 has movedto the right of the user 100, so the beamforming direction 608 is movedin a similar direction, being responsively changed to an updatedbeamforming direction 608. Though the beamforming direction 608 ischanged, its angular spread 610 is such that the HMD 102 is stilllocated within the main lobe, so as to maintain the wireless connectionwith the HMD, as the HMD has not actually moved to a new location yet.It will be appreciated that the beamforming direction has beenpredictively moved based on changes in the user's gaze direction. Whilethe HMD's location has not changed, the beamforming direction may bepredictively adjusted, but within a range that maintains the HMD 102within the angular spread of the transceiver 110.

At FIG. 6C, the user 100 has further moved his gaze direction to a gazedirection 612, by for example, additionally rotating his head. The userthen moves to a new location indicated by ref. 614. As the user 100moves, the beamforming direction of the transceiver is predictivelymoved to the direction 616, so as to maintain a strong wirelessconnection.

In some implementations, the gaze direction of the user (and/or changesthereof) is another factor that can be considered for purposes ofpredicting a future location of the HMD. The gaze direction can beweighted in combination with the additionally described factors fordetermining a predicted future location, and the beamforming directioncan be adjusted accordingly. Furthermore, in additional implementations,the gaze direction of the user can be applied to affect the beamformingangular spread.

In some implementations, the location of the HMD can be tracked overtime, and a distribution of the locations of the HMD within aninteractive environment can be determined. Future locations of the HMDcan be determined, at least in part, based on the historical locationdistribution of the HMD.

FIG. 7 illustrates an overhead view of a room 700 showing locationdistribution of an HMD, in accordance with implementations of thedisclosure. The room 700 defines an interactive real environment inwhich the HMD is operated by the user, and in which the camera 108 andthe transceiver 110 are disposed. The lines 704 a-e and 706 a-d areisometric location distribution lines based on historical locations ofthe HMD in the room 700. That is, the locations of the HMD duringinteractivity have been tracked over time, e.g. by recording thelocation of the HMD at periodic intervals, and the distribution of thelocations in the room 700 are such that the density (number ofoccurrences per unit area) or frequency or probability of occurrence isthe same or approximately the same along a given one of the lines 704a-e or 706 a-d. In the illustrated implementation, the highest isometricvalue illustrated is that of the lines 704 a and 706 a, with diminishingvalues for the lines 704 b, c, and d, as well as for lines 706 b, c, andd. In the illustrated implementation, the line 704 e represents thelowest isometric value that is illustrated.

It will be appreciated that the regions 708 and 710 exhibit the highestdistribution density of locations for the HMD. In other words, the HMDhas a statistically higher probability of being located in a unit areaof the regions 708 and 710 versus other being located in a unit area ofother regions of the room 700. In the illustrated implementation, acouch/chair 702 is shown in the room 700. The region 710 and surroundingregions correspond to a centrally seated location on the couch 702, asthe user may spend significant amounts of time using the HMD whileseated on the couch 702. The region 708 and surrounding regions arefront of the couch, and thus may indicate regions where the user isstanding in front of the couch while using the HMD.

In some implementations, the location distribution is utilized as afactor for determining the predicted future location of the HMD. Forexample, a probability or weight can be determined as a function oflocation that is indicative of the likelihood of the HMD being locatedat that location, and this can be used as a factor for determining thepredicted future location of the HMD.

In a related implementation, for a given interactive application, HMDlocation/movement patterns across a plurality of users can bedetermined, for example by recording location/movement information for aplurality of HMD's and uploading such information to a server forprocessing and analysis. The location/movement information is correlatedto the state of the interactive application, and thus HMDlocation/movement patterns for a given state of the interactiveapplication (e.g. at a particular temporal or geographical locationwithin a virtual environment defined by the interactive application) canbe determined. This can provide crowd-sourced data regarding HMDlocation and movement for specific application states, which can beutilized to predict future locations and movements of a particularuser's HMD during interaction with the interactive application.

FIG. 8 conceptually illustrates the use of a prediction model todetermine beamforming parameters, in accordance with implementations ofthe disclosure. The prediction model 800 is configured to predict afuture location and/or movement (e.g. velocity, acceleration) of the HMDusing one or more inputs.

By way of example, such inputs can include any of the following: motiondata 808 (e.g. velocity (direction and speed), acceleration, rotation,etc.), location data 810 (e.g. 3D coordinates, relative locationinformation, historical location information, etc.), gaze direction 812,user biometrics 814 (e.g. height, weight, heart rate, respiration, pupildilation, etc.), user profile/history (e.g. user preferences, usermovement/gesture patterns, etc.), and application state 818 (e.g.application variable states, virtual object states, etc.).

Based on the output of the prediction model, beamforming parameters ofthe transceiver are adjusted (ref. 802), which can include adjustment ofthe direction and/or angular spread of the main lobe. It will beappreciated that the beamforming of the transceiver is predictivelyadjusted so that the beamforming adjustments can occur simultaneous withor even prior to the actual movements of the HMD, so as to ensure thatthe HMD remains within the beamforming main lobe and is provided with aconsistently strong wireless connection.

At operation 804, feedback data can be processed to evaluate theeffectiveness of the beamforming adjustments and/or the predictionmodel's accuracy. In some implementations, the feedback data includessignal quality measurements taken by the HMD indicating the quality ofthe wireless signal received by the HMD from the transceiver. By way ofexample, such signal quality measurements can include signal strength,signal-to-noise ratio, bandwidth, errors, or other measures of thequality of the wireless signal transmitted by the transceiver andreceived by the HMD. By evaluating the signal quality of thetransceiver, the effectiveness of the beamforming adjustments and/or theaccuracy of the prediction model can be evaluated.

In some implementations, the feedback data includes location and/ormovement data indicating the actual locations and/or movements of theHMD, which can be compared to predicted locations/movements generated bythe prediction model, to evaluate the accuracy of the prediction model.

Based on the above, then at operation 806, the prediction model 800 canbe adjusted to improve its accuracy. In some implementations, machinelearning techniques can be applied to improve the prediction model.

FIG. 9 illustrates a method for adjusting beamforming parameters using apredicted future location, in accordance with implementations of thedisclosure. At method operation 900, images of a real-world interactiveenvironment including the HMD are captured by a camera. At methodoperation 902, inertial movements of the HMD are sensed by one or moreinertial sensors of the HMD. At method operation 904, the currentlocation of the HMD is determined based at least in part on one or bothof the sensed inertial movements of the HMD and the captured images ofthe HMD.

At method operation 906, a motion vector is generated based at least inpart on one or both of the sensed inertial movements of the HMD and thecaptured images of the HMD. At method operation 908, a future locationof the HMD is predicted using the motion vector and the current locationof the HMD. At method operation 910, one or more beamforming parametersof the transceiver, such as direction and/or angular spread, areadjusted based on the predicted future location of the HMD.

Though in the present disclosure, implementations have generally beendescribed with reference to predicting a future location of the HMD andsteering an RF beamforming direction towards the predicted futurelocation, it should be appreciated that in some implementations, aspecific future location is not necessarily determined. But rather, theadjustment of the beamforming direction in a predictive manner isachieved based on the various input parameters without specificallydetermining or identifying a particular future location. It will beappreciated that the beamforming direction in such implementations willbe predictively steered in a manner based on the inputs that would betowards a predicted future location if such was determined.

FIG. 10 conceptually illustrates a system for providing wirelesscommunication between a computer and a HMD, in accordance withimplementations of the disclosure. The computer 106 is connected to acamera 108 and a transceiver 110. As noted, the camera 108 and thetransceiver 110 may be part of the same device in some implementations,or separate devices in other implementations. The camera 108 includes acontroller 1026 that is configured to process instructions received fromthe computer 106 to control the camera's operating parameters, e.g.aperture, sensor gain, etc. The transceiver 110 includes a controller1028 that is configured to process instructions from the computer 106 tocontrol the operation of the transceiver 110 including control of thetransceiver's transmitter 1030 and receiver 1032. It will be appreciatedthat the transmitter 1030 and receiver 1032 can be configured to effectbeamforming in accordance with the principles of the present disclosure.

Broadly speaking the computer 106 executes an interactive application1016 (e.g. a video game) to generate video data (including image andaudio data) that is wirelessly transmitted to the HMD 102 for renderingto the display 1048 of the HMD 102. The beamforming direction and/orspread of the transceiver 110 are adjusted so as to maintain wirelesscoverage and directionality towards the HMD. The HMD includes variousinertial sensors 1038, for example including one or more accelerometers1040, gyroscopes 1042, and magnetometers 1044. Data processed from theinertial sensors 1038 is communicated by the HMD to the computer 106,via transmission from the HMD's transceiver 1034 to the transceiver 110.The computer 106 includes sensor data processor 1000 that is configuredto process the inertial sensor data from the HMD, e.g. to determine oridentify movements of the HMD.

The camera 108 is configured to capture images of the interactive realenvironment in which the user operates the HMD. The captured images bythe camera 108 are processed by the image analyzer 1002, e.g. toidentify the HMD, such as by identifying lights 1046 of the HMD 102.

Tracking logic 1004 is configured to further analyze, and identifyand/or quantify the location, orientation, and/or movement of the HMD.To this end a location analyzer 1006 is configured to determine thelocation of the HMD based on the inertial sensor data and the capturedimage data. An orientation analyzer is configured to determine theorientation of the HMD based on the inertial sensor data and thecaptured image data. A motion analyzer is configured to determine themotion of the HMD based on the inertial sensor data and the capturedimage data.

Prediction logic 1018 uses a model to predict a future location and/ormovement of the HMD 102 based on various inputs such as theaforementioned location, orientation and movement of the HMD 102. Insome implementations, the prediction logic 1018 uses additional inputssuch as user settings 1014 or information from the interactiveapplication 1016. For example, the interactive application 1016 mayprovide information regarding future expected locations or movements ofthe HMD, based on the current state of the interactive application. Abeamforming processor 1020 is configured to determine beamformingparameters and adjustments thereto, based on the predicted futurelocations and/or movements of the HMD. A direction processing module1022 is configured to determine the beamforming direction, andadjustments thereto, of the transceiver 110. A spread processing module1024 is configured to determine the angular spread, and adjustmentsthereto, of the transceiver 110. The updated beamforming parameters arecommunicated to the controller 1028 of the transceiver 110, whicheffects adjustment of the parameters of the transceiver, such assteering/updating the beamforming direction to an updated direction,and/or updating the angular spread.

In some implementations, the HMD 102 includes a signal analyzer 1036that is configured to evaluate the quality of the signal received fromthe transceiver 110. For example, signal analyzer 1036 may analyze thewireless signal from the transceiver 110 to determine its signalstrength. This information can be provided back to the computer 106 asfeedback, to enable evaluation of whether a strong signal is beingmaintained and the predictive adjustment of beamforming direction andangular spread is effective. In some implementations, the feedback datais provided via a separate communication channel and/or a separatecommunication protocol/context than that utilized for the transmissionof the video data to the HMD 102. For example, in some implementations,the feedback data is transmitted over the network 112 from the HMD tothe computer 106 (rather than being transmitted via the transceiver110). By way of example, the network 112 may include a wireless routeror other wireless networking device through which the HMD 102 wirelesslyaccesses the network 112. The computer 106 may also access the network106 through either a wired or wireless connection.

The use of an alternate communications protocol/context for purposes ofproviding the feedback data is beneficial in case wireless connectionvia the transceiver 110 is lost, in which case an alternate path forcommunication back to the computer 106 is possible. It will beappreciated that the bandwidth requirement for the transmission offeedback data, and other types of data, can be significantly less thanthat required for transmission of video data. Thus, transmission of thefeedback data over a communications context with less bandwidth (e.g.than that used to transmit video data to the HMD), for example aconventional WiFi network connection, can be sufficient for suchpurposes.

In some implementations, the transmission of feedback data occurs over aseparate frequency band than that used for the wireless transmission ofvideo data to the HMD. For example, the video data may be transmitted tothe HMD over a 60 GHz frequency band, whereas the feedback data istransmitted over different frequency band, e.g. a 2.4 GHz or 5 GHz band.It will be appreciated that in such implementations, the transmitter1030 of the transceiver 110 and the corresponding receiver of the HMD'stransceiver 1034 are configured to operate at 60 GHz, whereas thereceiver 1032 of the transceiver 110 and the corresponding transmitterof the HMD's transceiver 1034 are configured to operate at a differentfrequency band.

As has been noted, in some implementations beamforming is applied by thetransceiver 110 for both transmission and reception purposes. However,in some implementations, beamforming can be applied selectively by thetransceiver 110 for transmission only, while no beamforming is appliedfor reception. In this manner, communication from the HMD back to thetransceiver is more likely to be maintained even if transmission to theHMD is compromised or lost (e.g. due to failure of the main lobe toadequately track the HMD). In other implementations, beamforming can beapplied in different ways for transmission versus reception. Forexample, the angular spread of the beamforming for reception by thetransceiver 110 may be configured to be greater than the angular spreadof the beamforming for transmission by the transceiver 110. This canafford greater signal stability for receiving communication from the HMD(versus transmission to the HMD) while still providing some benefit interms of reception directionality.

In still further implementations, the quality of signal reception by thetransceiver 110 can serve as additional feedback data that is indicativeof whether the beamforming direction of the transceiver is beingeffectively steered towards the HMD.

Implementations of the disclosure employ beamforming as a signalprocessing technique to achieve directional signal transmission and/orreception. Beamforming technology entails operation of a phased array oftransmission or reception elements to purposely produce constructiveinterference in a desired direction and over a desired angular width.Beamforming can be used to achieve spatial selectivity for bothtransmission and reception. Broadly speaking, transmission beamformingentails control of the phase and relative amplitude of the signal ateach of a plurality of spatially separated antennas, whereas receptionbeamforming entails combining signals received from such antennas thathave been phase and amplitude adjusted. A basic discussion ofbeamforming can found with reference to “A Primer on DigitalBeamforming,” Toby Haynes, Spectrum Signal Processing, Mar. 26, 1998(http://www.spectrumsignal.com/publications/beamform_primer.pdf), thedisclosure of which is incorporated by reference.

Though implementations have generally been described with reference touse of inertial data and captured image data for purposes of determininglocation and movement of the HMD, it should be appreciated that theprinciples of the present disclosure can be applied with any knownmethod for determining location/orientation and/or movement of an HMD.For example, in some implementations, the HMD includes one or moreoutward facing cameras which can be utilized for movement and positiontracking, e.g. using simultaneous localization and mapping (SLAM)techniques as are known in the art. In some implementations,recognizable objects (e.g. emitters (e.g. RF, IR, visible spectrum,laser, ultrasonic, magnetic, etc.), lights, reflective objects, tags,shaped objects, patterns, etc.) can be positioned in the localenvironment to assist in such tracking. Such objects can be detected byappropriate sensors mounted on the HMD (e.g. camera, photo sensingdiode, magnetic sensor, microphone, etc.). It will be appreciated thatsuch sensors can include one or more sensors distributed about the HMD,or an array of sensors in a predefined configuration that can beoperated in concert to enable localization and tracking of the HMD. Anyknown method for localization and tracking of the HMD can be applied toenable predictive RF beamforming in accordance with the principles ofthe present disclosure, to enable a fully wirelessly operated HMD. Allsuch implementations are not described in detail herein, but will bereadily apparent to those skilled in the art and understood as part ofthe present disclosure.

FIG. 11A is a schematic diagram showing components of a beamformingtransmitter, such as the transmitter 1030 of the transceiver 110, inaccordance with implementations of the disclosure. An encoder 1100 isconfigured to receive and encode information for wireless transmission(e.g. video data for transmission to the HMD). The encoder 1100 mayformat or otherwise process the information for transmission, e.g.performing block encoding, compression, adding redundancy for errorreduction, etc. A modulator 1102 transforms the encoded data into awaveform, for example by mapping binary digits to a carrier frequency(e.g. pulse amplitude modulation (PAM), phase-shift keying (PSK), etc.).In some implementations, a carrier frequency is generated by a carrieroscillator 1104. Though not specifically shown, in some implementations,the waveform generated by the modulator can be frequency upconvertedand/or amplified.

The waveform is provided to a beamformer 1106, which feeds the waveformin parallel to a plurality of amplitude adjusters 1108 a-d, and to aplurality of phase shifters 1110 a-d. The amplitude adjusters and phaseshifters enable individual adjustment/tuning of the amplitude and phaseof the waveform for each antenna 1116 a-d of an antenna array 1114.Corresponding amplifiers 1112 a-d are provided to amplify the adjustedwaveform for transmission via the antennas 1116 a-d. The antennas 1116a-d of the antenna array 1114 are spatially arranged in a predefinedconfiguration. As noted, the transmission of the phase and amplitudeadjusted signals from the antennas of the antenna array produces awavefront having a pattern of constructive and destructive interferencethat produces the desired beamforming effect.

FIG. 11B is a schematic diagram showing components of a beamformingreceiver, such as the receiver 1032 of the transceiver 110, inaccordance with implementations of the disclosure. An antenna array 1120includes a plurality of antennas 1122 a-d. The signals received by theantenna array 1120 are fed to a beamformer 1124, which individuallyadjusts, for each antenna, the phase and amplitude of the receivedsignal via a plurality of phase adjusters 1126 a and amplitude adjusters1128 a. The adjusted signals are then combined via a combiner 1130,which may also amplify the combined signal. Though not specificallyshown, in some implementations, the combined signal can be frequencydownconverted and/or separately amplified.

A demodulator 1132 demodulates the combined signal to extract theencoded data, and a decoder 1134 decodes the encoded data to extract theoriginal information.

In some implementations, the antenna array 1114 (transmitter antennaarray) and the antenna array 1120 (receiver antenna array) are separatedevices. However, in other implementations, the antenna arrays 1114 and1120 are the same device, with, for example, a diplexer configured todivert transmission and reception signals appropriately. In variousimplementations, the antenna arrays may be microstrip/patch antennaarrays or other types of antenna arrays having a plurality of antennaspositioned in a predefined configuration to enable beamforming inaccordance with the principles of the present disclosure. Patch antennasas are known in the art may have tens to hundreds of individual antennaelements.

In some implementations, wireless communication in accordance with theprinciples of the present disclosure (e.g. for transmission of videodata to an HMD) occurs over a 60 GHz frequency band. In someimplementations, wireless communication takes place over other frequencybands, and may further utilize a combination of different frequencybands.

FIG. 12A conceptually illustrates a HMD having a plurality of antennaarrays, in accordance with implementations of the disclosure. As shown,the HMD includes an antenna array 1200 positioned at the front of theHMD 102, an antenna array 1202 positioned at the top of the HMD 102, anantenna array 1206 positioned at a side of the HMD 102, and an antennaarray 1208 positioned at the rear of the HMD 102. The antenna arrays areconnected to a selector 1210 that governs which of the antenna arrays isactive for purposes of signal reception and/or signal transmission bythe transceiver 1034. As the user 100 moves in the interactive realenvironment, the HMD's location and orientation may change, therebychanging which of the antenna arrays is optimally positioned. In someimplementations, the optimally positioned antenna array may be theantenna array that is nearest to the transceiver or which offers thebest line-of-sight to the transceiver. Accordingly, the selector 1210can be configured to switch between the various antenna arrays,selecting the one that is most optimally positioned. In someimplementations, the selector 1210 is configured to continuously measurethe reception signal strength from each of the antenna arrays 1200,1202, 106, and 1208, and determine which provides the highest signalstrength, and if necessary, then switch from using a current one of theantenna arrays to using the antenna array that provides the highestsignal strength.

Shown at ref. 1204 is an expanded representation of one antenna array.Each antenna array can include multiple individual antenna elements1205.

FIGS. 12B, 12C, and 12D illustrate overhead views of an HMD in aninteractive real environment, illustrating switching of active antennaarrays on an HMD, in accordance with implementations of the disclosure.At FIG. 12B, the front of HMD 102 is facing towards the transceiver 110.In accordance with implementations of the disclosure, the transceiver110 has a beamforming direction 1220 that is directed towards theantenna array 1200, which is the currently active antenna array of theHMD, from which received signals are processed to extract/decode videodata for rendering on the HMD. The additional antenna arrays 1206 a,1206 b, and 1208 are currently in an inactive state, meaning thatsignals received from these antenna arrays are not specificallyprocessed for video rendering as is the case for the antenna array 1200.However, the signals of the antenna arrays 1206 a/b and 1208 may stillbe monitored to, for example, determine their signal strength todetermine which of the antenna arrays is optimally positioned at a givenmoment.

At FIG. 12C the HMD has rotated in a clockwise direction, thus movingthe antenna array 1200. The transceiver is accordingly adjusted to havea beamforming direction 1222 that is towards the antenna array 1200, andmay have been predictively steered in accordance with the principlesdiscussed herein. The antenna array 1200 remains as the active antennaarray, while the others are inactive.

However, at FIG. 12D, the HMD 102 has rotated to a point wherein theantenna array 1206 a is now the nearest, and provides the mostunobstructed line-of-sight, to the transceiver 110. Therefore, theactive antenna array is switched from the antenna array 1200 to theantenna array 1206 a. Additionally, the beamforming direction of thetransceiver is redirected towards the newly active antenna array 1206 ainstead of the array 1200.

It will be appreciated that in some implementations, the orientation ofthe HMD in the interactive environment (e.g. relative to thetransceiver) can be determined using the inertial data and capturedimage data as previously described. The orientation of the HMD can thenbe utilized to determine which of the antenna arrays is most optimal forsignal reception by the HMD. Additionally, the presently describedantenna switching scheme can be performed in a predictive manner, suchthat antenna arrays are activated or deactivated based on predictedfuture orientations of the HMD.

With reference to FIG. 13, a diagram illustrating components of ahead-mounted display 102 is shown, in accordance with an embodiment ofthe disclosure. The head-mounted display 102 includes a processor 1300for executing program instructions. A memory 1302 is provided forstorage purposes, and may include both volatile and non-volatile memory.A display 1304 is included which provides a visual interface that a usermay view. A battery 1306 is provided as a power source for thehead-mounted display 102. A motion detection module 1308 may include anyof various kinds of motion sensitive hardware, such as a magnetometer1310, an accelerometer 1312, and a gyroscope 1314.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 1312 are used toprovide the direction of gravity, which gives an absolute reference fortwo angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the head-mounted display. In one embodiment, threemagnetometers 1310 are used within the head-mounted display, ensuring anabsolute reference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 1312 is used together with magnetometer 1310to obtain the inclination and azimuth of the head-mounted display 102.

In some implementations, the magnetometers of the head-mounted displayare configured so as to be read during times when electromagnets inother nearby devices are inactive.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 1314 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. This requires resettingthe gyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 1316 is provided for capturing images and image streams of areal environment. More than one camera may be included in thehead-mounted display 102, including a camera that is rear-facing(directed away from a user when the user is viewing the display of thehead-mounted display 102), and a camera that is front-facing (directedtowards the user when the user is viewing the display of thehead-mounted display 102). Additionally, a depth camera 1318 may beincluded in the head-mounted display 102 for sensing depth informationof objects in a real environment.

The head-mounted display 102 includes speakers 1320 for providing audiooutput. Also, a microphone 1322 may be included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The head-mounted display 102 includestactile feedback module 1324 for providing tactile feedback to the user.In one embodiment, the tactile feedback module 1324 is capable ofcausing movement and/or vibration of the head-mounted display 102 so asto provide tactile feedback to the user.

LEDs 1326 are provided as visual indicators of statuses of thehead-mounted display 102. For example, an LED may indicate batterylevel, power on, etc. A card reader 1328 is provided to enable thehead-mounted display 102 to read and write information to and from amemory card. A USB interface 1330 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the head-mounted display 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe head-mounted display 102.

A WiFi module 1332 is included for enabling connection to the Internetor a local area network via wireless networking technologies. Also, thehead-mounted display 102 includes a Bluetooth module 1334 for enablingwireless connection to other devices. A communications link 1336 mayalso be included for connection to other devices. In one embodiment, thecommunications link 1336 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1336 mayutilize any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1338 are included to provide an input interfacefor the user. Any of various kinds of input interfaces may be included,such as buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 1340 may be included in head-mounted display 102for facilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 1342 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 1342 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin.

A video input 1344 is configured to receive a video signal from aprimary processing computer (e.g. main game console) for rendering onthe HMD. In some implementations, the video input is an HDMI input.

The foregoing components of head-mounted display 102 have been describedas merely exemplary components that may be included in head-mounteddisplay 102. In various embodiments of the disclosure, the head-mounteddisplay 102 may or may not include some of the various aforementionedcomponents. Embodiments of the head-mounted display 102 may additionallyinclude other components not presently described, but known in the art,for purposes of facilitating aspects of the present disclosure as hereindescribed.

FIG. 14 is a block diagram of a Game System 1400, according to variousembodiments of the disclosure. Game System 1400 is configured to providea video stream to one or more Clients 1410 via a Network 1415. GameSystem 1400 typically includes a Video Server System 1420 and anoptional game server 1425. Video Server System 1420 is configured toprovide the video stream to the one or more Clients 1410 with a minimalquality of service. For example, Video Server System 1420 may receive agame command that changes the state of or a point of view within a videogame, and provide Clients 1410 with an updated video stream reflectingthis change in state with minimal lag time. The Video Server System 1420may be configured to provide the video stream in a wide variety ofalternative video formats, including formats yet to be defined. Further,the video stream may include video frames configured for presentation toa user at a wide variety of frame rates. Typical frame rates are 30frames per second, 60 frames per second, and 120 frames per second.Although higher or lower frame rates are included in alternativeembodiments of the disclosure.

Clients 1410, referred to herein individually as 1410A, 1410B, etc., mayinclude head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, Clients 1410 are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the Client. The video streams may be presented to the user on adisplay integral to Client 1410 or on a separate device such as amonitor or television. Clients 1410 are optionally configured to supportmore than one game player. For example, a game console may be configuredto support two, three, four or more simultaneous players. Each of theseplayers may receive a separate video stream, or a single video streammay include regions of a frame generated specifically for each player,e.g., generated based on each player's point of view. Clients 1410 areoptionally geographically dispersed. The number of clients included inGame System 1400 may vary widely from one or two to thousands, tens ofthousands, or more. As used herein, the term “game player” is used torefer to a person that plays a game and the term “game playing device”is used to refer to a device used to play a game. In some embodiments,the game playing device may refer to a plurality of computing devicesthat cooperate to deliver a game experience to the user. For example, agame console and an HMD may cooperate with the video server system 1420to deliver a game viewed through the HMD. In one embodiment, the gameconsole receives the video stream from the video server system 1420, andthe game console forwards the video stream, or updates to the videostream, to the HMD for rendering.

Clients 1410 are configured to receive video streams via Network 1415.Network 1415 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 1410 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 1410 may, but are not required to, further include systemsconfigured for modifying received video. For example, a Client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 1410 may be configured to receive various types ofvideo frames, such as I-frames, P-frames and B-frames, and to processthese frames into images for display to a user. In some embodiments, amember of Clients 1410 is configured to perform further rendering,shading, conversion to 3-D, or like operations on the video stream. Amember of Clients 1410 is optionally configured to receive more than oneaudio or video stream. Input devices of Clients 1410 may include, forexample, a one-hand game controller, a two-hand game controller, agesture recognition system, a gaze recognition system, a voicerecognition system, a keyboard, a joystick, a pointing device, a forcefeedback device, a motion and/or location sensing device, a mouse, atouch screen, a neural interface, a camera, input devices yet to bedeveloped, and/or the like.

The video stream (and optionally audio stream) received by Clients 1410is generated and provided by Video Server System 1420. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. As usedherein, the term “video frames” is used to refer to frames includingpredominantly information that is configured to contribute to, e.g. toeffect, the images shown to the user. Most of the teachings herein withregard to “video frames” can also be applied to “audio frames.”

Clients 1410 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 1410. The received gamecommands are communicated from Clients 1410 via Network 1415 to VideoServer System 1420 and/or Game Server 1425. For example, in someembodiments, the game commands are communicated to Game Server 1425 viaVideo Server System 1420. In some embodiments, separate copies of thegame commands are communicated from Clients 1410 to Game Server 1425 andVideo Server System 1420. The communication of game commands isoptionally dependent on the identity of the command Game commands areoptionally communicated from Client 1410A through a different route orcommunication channel that that used to provide audio or video streamsto Client 1410A.

Game Server 1425 is optionally operated by a different entity than VideoServer System 1420. For example, Game Server 1425 may be operated by thepublisher of a multiplayer game. In this example, Video Server System1420 is optionally viewed as a client by Game Server 1425 and optionallyconfigured to appear from the point of view of Game Server 1425 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 1420 and Game Server 1425 optionally occursvia Network 1415. As such, Game Server 1425 can be a prior artmultiplayer game server that sends game state information to multipleclients, one of which is game server system 1420. Video Server System1420 may be configured to communicate with multiple instances of GameServer 1425 at the same time. For example, Video Server System 1420 canbe configured to provide a plurality of different video games todifferent users. Each of these different video games may be supported bya different Game Server 1425 and/or published by different entities. Insome embodiments, several geographically distributed instances of VideoServer System 1420 are configured to provide game video to a pluralityof different users. Each of these instances of Video Server System 1420may be in communication with the same instance of Game Server 1425.Communication between Video Server System 1420 and one or more GameServer 1425 optionally occurs via a dedicated communication channel. Forexample, Video Server System 1420 may be connected to Game Server 1425via a high bandwidth channel that is dedicated to communication betweenthese two systems.

Video Server System 1420 comprises at least a Video Source 1430, an I/ODevice 1445, a Processor 1450, and non-transitory Storage 1455. VideoServer System 1420 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 1430 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments, Video Source 1430 includes a video game engine andrendering logic. The video game engine is configured to receive gamecommands from a player and to maintain a copy of the state of the videogame based on the received commands. This game state includes theposition of objects in a game environment, as well as typically a pointof view. The game state may also include properties, images, colorsand/or textures of objects. The game state is typically maintained basedon game rules, as well as game commands such as move, turn, attack, setfocus to, interact, use, and/or the like. Part of the game engine isoptionally disposed within Game Server 1425. Game Server 1425 maymaintain a copy of the state of the game based on game commands receivedfrom multiple players using geographically disperse clients. In thesecases, the game state is provided by Game Server 1425 to Video Source1430, wherein a copy of the game state is stored and rendering isperformed. Game Server 1425 may receive game commands directly fromClients 1410 via Network 1415, and/or may receive game commands viaVideo Server System 1420.

Video Source 1430 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 1455. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 1410. Forexample, the raw video may be encoded according to an Adobe Flash®standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith,MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like. Theencoding process produces a video stream that is optionally packaged fordelivery to a decoder on a remote device. The video stream ischaracterized by a frame size and a frame rate. Typical frame sizesinclude 800×600, 1280×720 (e.g., 720p), 1024×768, although any otherframe sizes may be used. The frame rate is the number of video framesper second. A video stream may include different types of video frames.For example, the H.264 standard includes a “P” frame and a “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In alternative embodiments Video Source 1430 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream, optionally includes both rendered images andimages recorded using a still or video camera. Video Source 1430 mayalso include storage devices configured to store previously recordedvideo to be included in a video stream. Video Source 1430 may alsoinclude motion or positioning sensing devices configured to detectmotion or position of an object, e.g., person, and logic configured todetermine a game state or produce video-based on the detected motionand/or position.

Video Source 1430 is optionally configured to provide overlaysconfigured to be placed on other video. For example, these overlays mayinclude a command interface, log in instructions, messages to a gameplayer, images of other game players, video feeds of other game players(e.g., webcam video). In embodiments of Client 1410A including a touchscreen interface or a gaze detection interface, the overlay may includea virtual keyboard, joystick, touch pad, and/or the like. In one exampleof an overlay a player's voice is overlaid on an audio stream. VideoSource 1430 optionally further includes one or more audio sources.

In embodiments wherein Video Server System 1420 is configured tomaintain the game state based on input from more than one player, eachplayer may have a different point of view comprising a position anddirection of view. Video Source 1430 is optionally configured to providea separate video stream for each player based on their point of view.Further, Video Source 1430 may be configured to provide a differentframe size, frame data size, and/or encoding to each of Client 1410.Video Source 1430 is optionally configured to provide 3-D video.

I/O Device 1445 is configured for Video Server System 1420 to sendand/or receive information such as video, commands, requests forinformation, a game state, gaze information, device motion, devicelocation, user motion, client identities, player identities, gamecommands, security information, audio, and/or the like. I/O Device 1445typically includes communication hardware such as a network card ormodem. I/O Device 1445 is configured to communicate with Game Server1425, Network 1415, and/or Clients 1410.

Processor 1450 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 1420 discussedherein. For example, Processor 1450 may be programmed with softwareinstructions in order to perform the functions of Video Source 1430,Game Server 1425, and/or a Client Qualifier 1460. Video Server System1420 optionally includes more than one instance of Processor 1450.Processor 1450 may also be programmed with software instructions inorder to execute commands received by Video Server System 1420, or tocoordinate the operation of the various elements of Game System 1400discussed herein. Processor 1450 may include one or more hardwaredevice. Processor 1450 is an electronic processor.

Storage 1455 includes non-transitory analog and/or digital storagedevices. For example, Storage 1455 may include an analog storage deviceconfigured to store video frames. Storage 1455 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 1415 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, and/or the like. Storage 1455 is optionallydistributed among a plurality of devices. In some embodiments, Storage1455 is configured to store the software components of Video Source 1430discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 1420 optionally further comprises Client Qualifier1460. Client Qualifier 1460 is configured for remotely determining thecapabilities of a client, such as Clients 1410A or 1410B. Thesecapabilities can include both the capabilities of Client 1410A itself aswell as the capabilities of one or more communication channels betweenClient 1410A and Video Server System 1420. For example, Client Qualifier1460 may be configured to test a communication channel through Network1415.

Client Qualifier 1460 can determine (e.g., discover) the capabilities ofClient 1410A manually or automatically. Manual determination includescommunicating with a user of Client 1410A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 1460 isconfigured to display images, text, and/or the like within a browser ofClient 1410A. In one embodiment, Client 1410A is an HMD that includes abrowser. In another embodiment, client 1410A is a game console having abrowser, which may be displayed on the HMD. The displayed objectsrequest that the user enter information such as operating system,processor, video decoder type, type of network connection, displayresolution, etc. of Client 1410A. The information entered by the user iscommunicated back to Client Qualifier 1460.

Automatic determination may occur, for example, by execution of an agenton Client 1410A and/or by sending test video to Client 1410A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 1460. In various embodiments, the agent can find outprocessing power of Client 1410A, decoding and display capabilities ofClient 1410A, lag time reliability and bandwidth of communicationchannels between Client 1410A and Video Server System 1420, a displaytype of Client 1410A, firewalls present on Client 1410A, hardware ofClient 1410A, software executing on Client 1410A, registry entrieswithin Client 1410A, and/or the like.

Client Qualifier 1460 includes hardware, firmware, and/or softwarestored on a computer readable medium. Client Qualifier 1460 isoptionally disposed on a computing device separate from one or moreother elements of Video Server System 1420. For example, in someembodiments, Client Qualifier 1460 is configured to determine thecharacteristics of communication channels between Clients 1410 and morethan one instance of Video Server System 1420. In these embodiments theinformation discovered by Client Qualifier can be used to determinewhich instance of Video Server System 1420 is best suited for deliveryof streaming video to one of Clients 1410.

Embodiments of the present disclosure may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Thedisclosure can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that thedisclosure can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the disclosure are useful machineoperations. The disclosure also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The disclosure can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing disclosure has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the disclosure isnot to be limited to the details given herein, but may be modifiedwithin the scope and equivalents of the present disclosure.

What is claimed is:
 1. A method, comprising: receiving captured imageframes of an interactive environment in which a head-mounted display(HMD) is disposed; receiving inertial data processed from at least oneinertial sensor of the HMD; analyzing the captured image frames and theinertial data to determine a predicted future location of the HMD,wherein analyzing the captured image frames and the inertial dataincludes identifying movement of the HMD, the predicted future locationof the HMD being determined using the identified movement of the HMD,wherein analyzing the captured image frames includes examining changeswithin a sequence of said captured image frames to track the movement ofthe HMD, wherein analyzing the captured image frames includes processingthe captured image frames to identify and map one or more anchors in theinteractive environment and tracking the one or more anchors to enablethe tracking of the motion of the HMD, the one or more anchors includingone or more fixed objects in the interactive environment; using thepredicted future location of the HMD to adjust a beamforming directionof an RF transceiver towards the predicted future location of the HMD;adjusting an angular spread of the RF transceiver based on a speed of amovement of the HMD, wherein the angular spread increases withincreasing speed of the movement of the HMD.
 2. The method of claim 1,wherein identifying the movement of the HMD includes determining amotion vector of the HMD, the predicted future location of the HMD beingdetermined by applying the motion vector of the HMD to a currentlocation of the HMD.
 3. The method of claim 2, wherein a magnitude ofthe motion vector identifies the speed of the movement of the HMD, andwherein a direction of the motion vector identifies a direction of themovement of the HMD.
 4. The method of claim 2, wherein identifying themovement of the HMD includes identifying translational movement and/orrotational movement of the HMD; wherein determining the motion vectorincludes determining an acceleration of the translational movementand/or the rotational movement.
 5. The method of claim 1, wherein the RFtransceiver includes a phased array of RF emitters; wherein adjustingthe beamforming direction of the RF transceiver includes generatingtransceiver control data that is configured to cause adjustment of aphase or amplitude of at least one of the RF emitters of the phasedarray.
 6. The method of claim 1, wherein the at least one inertialsensor of the HMD includes one or more of an accelerometer, a gyroscope,or a magnetometer.
 7. The method of claim 1, wherein the HMD includes aplurality of lights; wherein analyzing the captured image framesincludes identifying one or more of the plurality of lights in thecaptured image frames.
 8. A system, comprising: a head-mounted display(HMD), the HMD having at least one inertial sensor configured togenerate inertial data; a camera configured to capture image frames ofan interactive environment in which the HMD is disposed; an RFtransceiver; a computer configured to analyze the captured image framesand the inertial data to determine a predicted future location of theHMD, and use the predicted future location of the HMD to adjust abeamforming direction of the RF transceiver towards the predicted futurelocation of the HMD, wherein analyzing the captured image frames and theinertial data includes identifying movement of the HMD, the predictedfuture location of the HMD being determined using the identifiedmovement of the HMD, wherein analyzing the captured image framesincludes examining changes within a sequence of said captured imageframes to track the movement of the HMD, wherein analyzing the capturedimage frames includes processing the captured image frames to identifyand map one or more anchors in the interactive environment and trackingthe one or more anchors to enable the tracking of the motion of the HMD,the anchors including one or more fixed objects in the interactiveenvironment; wherein the computer is further configured to adjust anangular spread of the RF transceiver based on a radial distance of theHMD from the RF transceiver.
 9. The system of claim 8, whereinidentifying movement of the HMD includes determining a motion vector ofthe HMD, the predicted future location of the HMD being determined byapplying the motion vector of the HMD to a current location of the HMD.10. The system of claim 9, wherein a magnitude of the motion vectoridentifies a speed of the movement of the HMD, and wherein a directionof the motion vector identifies a direction of the movement of the HMD.11. The system of claim 10, wherein the computer is further configuredto adjust an angular spread of the RF transceiver based on the speed ofthe movement of the HMD.
 12. The system of claim 11, wherein the angularspread increases with increasing speed of the movement of the HMD. 13.The system of claim 9, wherein identifying movement of the HMD includesidentifying translational movement and/or rotational movement of theHMD; wherein determining the motion vector includes determining anacceleration of the translational movement and/or the rotationalmovement.
 14. The system of claim 8, wherein the RF transceiver includesa phased array of RF emitters; wherein adjusting the beamformingdirection of the RF transceiver includes generating transceiver controldata that is configured to cause adjustment of a phase or amplitude ofat least one of the RF emitters of the phased array.
 15. The system ofclaim 8, wherein the at least one inertial sensor of the HMD includesone or more of an accelerometer, a gyroscope, or a magnetometer.
 16. Thesystem of claim 8, wherein the HMD includes a plurality of lights;wherein analyzing the captured image frames includes identifying one ormore of the plurality of lights in the captured image frames.
 17. Amethod, comprising: receiving captured image frames of an interactiveenvironment in which a head-mounted display (HMD) is disposed; receivinginertial data processed from at least one inertial sensor of the HMD;analyzing the captured image frames and the inertial data to determine apredicted future location of the HMD, wherein analyzing the capturedimage frames and the inertial data includes identifying movement of theHMD, the predicted future location of the HMD being determined using theidentified movement of the HMD, wherein analyzing the captured imageframes includes examining changes within a sequence of said capturedimage frames to track the movement of the HMD, wherein analyzing thecaptured image frames includes processing the captured image frames toidentify and map one or more anchors in the interactive environment andtracking the one or more anchors to enable the tracking of the motion ofthe HMD, the one or more anchors including one or more fixed objects inthe interactive environment; using the predicted future location of theHMD to adjust a beamforming direction of an RF transceiver towards thepredicted future location of the HMD; adjusting an angular spread of theRF transceiver based on a transmission data rate of the RF transceiver.